Skip to content
This repository was archived by the owner on Mar 15, 2022. It is now read-only.

Compared performance of pure-Java references vs. Java interop. #19

Closed
wants to merge 3 commits into from

Conversation

jdantonio
Copy link
Member

Performance Comparison -- Do Not Merge

In PR #17 we removed the pure-Java reference classes and replaced them with Ruby wrapper classes. Both implementations wrapped the same classes from java.lang.ref so I assumed the performance would be similar. It appears that I was wrong. Based on the speed test in this PR, the original implementation was much faster. Subsequently we should probably revert the change.

If anyone has some time, please verify my tests and findings.

~~~ Ruby version: 1.9.3
~~~ JRuby version: 1.7.18
~~~ Gem version: 1.0.5
       user     system      total        real
Benchmarking Ref::SoftReference...
   0.270000   0.010000   0.280000 (  0.114000)
Benchmarking Ref::WeakReference...
   0.080000   0.000000   0.080000 (  0.029000)

~~~ Ruby version: 1.9.3
~~~ JRuby version: 1.7.18
~~~ Gem version: 2.0.0.pre1
       user     system      total        real
Benchmarking Ref::SoftReference...
   1.020000   0.040000   1.060000 (  0.410000)
Benchmarking Ref::WeakReference...
   0.200000   0.020000   0.220000 (  0.165000)

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.21%) to 96.61% when pulling 22751c2 on performance-comparison into cd9191a on master.

@lucasallan
Copy link
Member

@jdantonio How many time did you run the test?

@jdantonio
Copy link
Member Author

@lucasallan Only a couple of times. It wasn't a comprehensive test. It's just something I knocked out this morning before starting work.

@lucasallan
Copy link
Member

I knew we would have a small performance impact - pure Java is faster than JRuby.

With that being said, I think we should stick with the "JRuby interop" version because it makes the gem easier to maintain and the performance hit is not something that would make someone not use the gem.

@lucasallan
Copy link
Member

~~~ Ruby version: 1.9.3
~~~ JRuby version: 1.7.18
~~~ Gem version: 2.0.0.pre1
       user     system      total        real
Benchmarking Ref::SoftReference...
   0.300000   0.000000   0.300000 (  0.274000)
Benchmarking Ref::WeakReference...
   0.220000   0.010000   0.230000 (  0.204000)
~~~ Ruby version: 1.9.3
~~~ JRuby version: 1.7.18
~~~ Gem version: 1.0.5
       user     system      total        real
Benchmarking Ref::SoftReference...
   0.080000   0.000000   0.080000 (  0.079000)
Benchmarking Ref::WeakReference...
   0.070000   0.000000   0.070000 (  0.078000)

@jdantonio
Copy link
Member Author

I was actually surprised by the difference in performance, but that may simply be my lack of understanding of JRuby internals. In both cases we're wrapping a pure-Java java.lang.ref.* class. In both cases we're putting a Ruby object inside the pure-Java ref class. In both cases we have to cross the barrier between the Ruby code and the Java code. Yet one is 4-5 times faster than the other. What's going on in the JRuby-interop version that makes it so much slower?

In this case I think the performance difference may outweigh the maintenance difference. The Java code is already written as are the build tasks. Since it's just a wrapper there should be no need to change the Java code unless Oracle adds new methods to the java.lang.ref.* classes. We can reasonably expect that code to be very stable. But I think we can also reasonably expect applications that use this gem to be performance-sensitive.

I'm curious to hear what others on the team think.

@lucasallan
Copy link
Member

Sounds good. I'll revert it and we should also remove it from the Roadmap.

@jdantonio
Copy link
Member Author

We don't have to revert yet. There is no rush. We can give the others a day or two to respond. Our teammates are mostly in Europe so there is a significant time difference.

@lucasallan
Copy link
Member

Oh yes, sorry! I forgot to mention that I'll wait the team.

@pitr-ch
Copy link
Member

pitr-ch commented Jan 31, 2015

We need to let the benchmark warm up, running it for much longer and with the --server flag getting all the optimizations.

@pitr-ch
Copy link
Member

pitr-ch commented Feb 6, 2015

Sorry for useless comment, I made it in a hurry. You've already did what I've suggested. Even though I've tried with more cycles 50_000_000 and with WARM_UPS = NUM because server JIT threshold is around 10k. Updated results are pushed.

I think it makes sense to revert.

@coveralls
Copy link

Coverage Status

Coverage remained the same at 96.83% when pulling 392056f on performance-comparison into cd9191a on master.

@jdantonio
Copy link
Member Author

I agree with reverting.

This also has me thinking about the wrapped objects in concurrent-ruby, such as AtomicBoolean and AtomicFixnum. I think we should experiment with pure-Java wrappers for those objects (following this pattern) and see if it makes a difference. If there is a similar performance difference we may want to adopt this pattern in that gem.

@pitr-ch
Copy link
Member

pitr-ch commented Feb 6, 2015

@jdantonio yeah, good point.

@lucasallan
Copy link
Member

I'll revert it tonight.

@jdantonio jdantonio closed this Feb 11, 2015
@jdantonio jdantonio deleted the performance-comparison branch February 11, 2015 04:26
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants